1. Field of the Invention
This invention relates to the fields of confocal and non-confocal imaging of large microscope specimens with particular emphasis on scanning beam fluorescence and photoluminescence imaging systems, including multi-photon fluorescence, spectrally-resolved fluorescence, and second and third harmonic imaging. Applications include imaging tissue specimens, genetic microarrays, protein arrays, tissue arrays, cells and cell populations, biochips, arrays of biomolecules, detection of nanoparticles, photoluminescence imaging of semiconductor materials and devices, Raman imaging, and many others.
2. Description of the Prior Art
FIG. 1 shows one embodiment of a prior art confocal scanning laser macroscope, as described in U.S. Pat. No. 5,760,951. In this embodiment, the incoming collimated laser beam 102 from laser 100 passes through a beam expander (comprised of lens 104 and lens 106), and is expanded to match the diameter of entrance pupil 112 of laser scan lens 114 (note—entrance pupil 112 as indicated on the figure simply indicates the position of the entrance pupil. A real stop is not usually placed at this position). Scanning mirror 110 deflects the beam in the X direction. Laser scan lens 114 focuses the beam to spot 116 on sample 118, mounted on microscope slide 120, and light reflected from or emitted by the sample is collected by laser scan lens 114, descanned by scanning mirror 110, and partially reflected by beamsplitter 108 into a confocal detection arm comprised of laser rejection filter 130, lens 132, pinhole 134, and detector 136. Detector 136 is located behind pinhole 134. Light reflected back from focused spot 116 on sample 118 passes through pinhole 134 and is detected, but light from any other point in the sample runs into the edges of the pinhole and is not detected. The scanning mirror is computer-controlled to raster the focused spot across the sample. At the same time, microscope slide 120, which is mounted on a computer-controlled, motor-driven scanning stage 122, moves slowly in the Y direction. The combination of rapid beam scanning across the sample while it is moved slowly in the perpendicular Y direction results in a raster-scan motion of focused-laser spot 116 across sample 118. A computer, represented by computer screen 140, is connected to the detector 136 to store and display a signal from the detector 136. The computer provides means for displaying and storing the signal from the detector. This confocal macroscope has properties similar to those of a confocal scanning laser microscope, except that the field of view of the microscope is much smaller.
FIG. 2 shows a second embodiment of a prior art confocal scanning laser macroscope for simultaneous imaging of two different fluorophores. This instrument uses a two-laser or other two-wavelength source of collimated light, with the source wavelengths chosen to match the excitation wavelengths of the two fluorophores. If more than two fluorophores are present, additional laser wavelengths and detection arms can be added, or a spectrally-resolved detector can be used in a single detection arm. When imaging fluorescent nanoparticles, a single laser source can be used with multiple detection arms, or with a spectrally-resolved detector. A collimated light beam 102 from two-wavelength source 200 is expanded by a beam expander comprised of lens 104 and lens 106, and passes through dichroic filters 208 and 210 on its way to scanning mirror 110. Scanning proceeds as it did in the macroscope described in FIG. 1. Here, light emitted from both fluorophores travels back toward the two detection arms, with light from one fluorophore reflected by dichroic filter 210 into the second detection arm, comprised of laser rejection filter 230, focusing lens 232, pinhole 234 (placed at the focal point of focusing lens 232 in this infinity-corrected system) and is detected by detector 236. Light from the other fluorophore passes through dichroic mirror 210 and is reflected by dichroic mirror 208 into the first detection arm comprised of laser rejection filter 130, focusing lens 132, pinhole 134 and detector 136. Each detector sends an electrical signal proportional to the intensity of the light detected to an A/D converter (not shown) where the intensity of light detected at each pixel position for each fluorophore is converted to a digital value that is stored in an image file. Although many other detectors can be used, we usually use detectors that are comprised of a photomultiplier tube and a preamplifier. One of the advantages of this instrument when imaging multiple fluorophores is the ability to separately adjust the gain of each detector depending on the fluorescence intensity of that fluorophore.
FIG. 3 shows a third embodiment of a prior art scanning laser macroscope that images in brightfield in addition to fluorescence. In order to more clearly illustrate the transmission brightfield optics, the scanning stage is not shown in this diagram, however a scanning stage like that shown in FIG. 1 is used in this instrument. In the instrument described in FIG. 3 the multiple-laser source 300 provides red, green and blue laser wavelengths for RGB brightfield imaging and for exciting fluorophores, as well as one or more additional laser sources that can be used for exciting additional fluorophores. Brightfield imaging is accomplished by collecting the light that passes through specimen 118 and microscope slide 120. A large-NA collection lens 302 directs the transmitted light toward RGB detector 304 for recording the brightfield image. The output of detector 304 is sent to Computer 140 (as shown in FIG. 1). Each of the three colours (red, green and blue) are digitized (usually using 8 bits for each colour), resulting in a 24-bit RGB image. White balance can be adjusted by changing the gain in the Red, Green and Blue channels, or after imaging by adjusting the image data file.
Several other embodiments of the macroscope are presently in use. These include instruments for fluorescence and photoluminescence (including spectrally-resolved) imaging (several other contrast mechanisms are also possible), instruments in which the specimen stage is stationary and the raster scan is provided by two scanning mirrors rotating about perpendicular axes, non-confocal versions, and other embodiments. A macroscope with fine focus adjustment is described in U.S. Pat. No. 7,218,446 B2, and versions for reflected-light, fluorescence, photoluminescence, multi-photon fluorescence, transmitted-light, and brightfield imaging are described. The combination of a scanning laser macroscope with a scanning laser microscope to provide an imaging system with a wide field of view and the high resolution capability of a microscope is described in U.S. Pat. No. 5,532,873.
Several other technologies are used for imaging large specimens at high resolution. With tiling microscopes, the image of a small area of the specimen is recorded with a digital camera (usually a CCD or CMOS camera), the specimen is moved with a computer-controlled microscope stage, an image of the adjacent area is recorded, and so on until a number of image tiles have been recorded that together cover the whole area of the specimen. These image tiles can be butted together, or overlapped and stitched using computer stitching algorithms, to form one image of the entire specimen. Such images may contain tiling artifacts, caused by focus changes between adjacent tiles, differences in illumination intensity across the field of view of the microscope, and microscope objectives that do not have a flat focal plane.
When tiling microscopes are used for fluorescence imaging, the areas surrounding each tile and the overlapping edges of adjacent tiles are exposed twice (and the corners four times) which can bleach some fluorophores. Exposure is adjusted by changing the exposure time for each tile. If multiple fluorophores are imaged, a different exposure time is required for each, so each fluorophore requires a separate image at each tile position. Multiple exposure of the specimen for imaging multiple fluorophores can also increase bleaching. After all tiles have been collected, considerable effort (both human and computer) is required to stitch the tiles together and correct each tile for illumination intensity and collection sensitivity changes across the field of view of the microscope (correction for variations in illumination intensity and collection sensitivity is sometimes called “field flattening”). Stitching tiles together is also complicated by distortion and curvature of field of the microscope objective. The distortion and curvature are maximized near the edges of the field of view (just where stitching of tiles occurs).
Strip scanning instruments are also used for imaging large specimens. In these instruments, a short line of white light (about 1 mm long) is focused on the sample from above, and a linear CCD detector with 1000 or 2000 pixels is placed below the sample to collect light from each pixel position in the illuminated line in the specimen. Three separate linear detectors with appropriate filters to pass red, green and blue light are used for RGB brightfield imaging. The sample is moved in the direction perpendicular to the illuminated line to scan a narrow strip across the width of a microscope slide. The entire slide can be imaged by imaging repeated strips and butting them together to create the final image. Another version of this technology uses three linear TDI (time delay integration) sensors which increases both sensitivity and imaging speed. In both of these instruments, exposure is varied by changing scan speed.
Fluorescence imaging requires sensitivity that is thousands of times greater than for brightfield imaging, making it difficult to use the present strip-scanning instruments for fluorescence imaging, since they were designed for red, green and blue image channels with gains set to provide proper white balance in the final image, and equal exposure time for each channel. In fluorescence imaging, white balance has no meaning, and fluorescence imaging also requires large differences in exposure from one fluorophore to another, making it very difficult to use a strip-scanning instrument for simultaneous imaging of multiple fluorophores. In addition, for excitation of multiple fluorophores, it is useful to be able to choose a particular laser wavelength and intensity for excitation of each fluorophore. White light excitation is appropriate for brightfield imaging, but does not work well for multiple fluorophores (since the illumination includes wavelengths that overlap the fluorescence wavelengths being detected), or for fluorophores excited by wavelengths outside the wavelength range of white light (a good example is DAPI, a common fluorophore excited in the near UV).
When the macroscope is used for fluorescence imaging, it has several advantages. Exposure for each fluorophore can be adjusted separately without changing scan speed by changing either laser intensity and/or detector gain (in the case of a detector comprised of a photomultiplier tube (pmt) followed by a preamplifier, both the pmt voltage (which changes pmt gain) and preamplifier gain can be changed). The ability to adjust the detection gain for each fluorophore separately allows the instrument to simultaneously collect multiple fluorophore images that are all correctly exposed. In addition, the appropriate laser wavelength can be provided to excite a chosen fluorophore, and the excitation wavelengths can be chosen so they do not overlap the detection wavelength ranges.
Challenges for Imaging Very Large Specimens in Fluorescence
When very large specimens are imaged in fluorescence or in brightfield, file sizes are very large, which makes it difficult and time-consuming to store, view, process, analyze and transmit the resulting image data sets. For example, with one micron pixels and 8 bits per pixel, imaging the entire area of a microscope slide (2.5×7 cm) results in a 1.875 Gpixel image. If this is a brightfield image, with 24 bits per pixel (RGB), the resulting file size is 5.625 GB. If the resolution is increased by a factor of two to 0.5 micron pixels, the file size increases by a factor of four to 22.5 GB. 0.25 micron pixel size results in a 90 GB file.
In fluorescence imaging, the fluorescence intensity is often measured with a dynamic range of either 12 or 16 bits per fluorophore and stored as 16-bit data sets, so a 12-bit or 16-bit fluorescence image with three fluorophores requires a file size twice that of the greyscale brightfield image just described. Scanners for large microscopy specimens presently use pixels as small as 0.25 microns and microscope slides up to 5×7 inches in size. This combination results in a file size of 1.05 TB, even with only 24 bits per pixel.
File size limitations in some operating systems mean these data sets have to be broken up into multiple files for storage. Lossless (and sometimes lossy) compression is sometimes used to reduce the file size. A pyramidal file structure is often used, so that a small area of the image can be viewed at high resolution without loading the entire image into RAM. Although these large images can be stored in a pyramidal file structure that will allow the user to zoom in and roam around without loading the whole image into RAM, many image processing operations require the entire image file to be accessed, and some require the entire file to be loaded into RAM. If it is necessary to transmit large images to another location for analysis or storage, large bandwidth is required and the transmission time is long. Using a 100 GB file as an example, and a fiber network capable of transferring 1000 Mbps, if we assume a file transfer rate of 100 MBps, a 100 GB file would take 1000 seconds to transfer (about 17 minutes). At a download speed of 1000 kBps (a common download speed for high-speed internet connections), such a file would take 27.8 hours to transfer.
Most image processing and analysis operations require the entire file to be opened (at a USB-II hard drive file transfer rate of 500 Mbps it will take 27 minutes just to open a 100 GB file). Some image analysis programs (like Photoshop) open two copies of the image in RAM so changes can be made and previewed without having to access the stored file for every operation. This limits the size of image that can practically be analyzed using these programs to less than half the RAM available in the computer.
Before scanning a large specimen in fluorescence, it is important to set the exposure time (in a tiling or strip-scanning microscope) or the combination of laser intensity, detector gain and scan speed (in a scanning laser macroscope or microscope) so that the final image will be properly exposed—in general it should not contain saturated pixels, but the gain should be high enough that the full dynamic range will be used for each fluorophore in the final image. Two problems must be solved to achieve this result—the exposure (or gain) must be estimated in advance for each fluorophore, and for simultaneous detection of multiple fluorophores the exposure time (or gain) must be adjusted separately for each detection channel before scanning. For strip-scanning instruments, where exposure time is set by changing the scan speed, simultaneous detection of multiple fluorophores is very difficult if different exposures are required for each fluorophore.